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This paper considers the design of binary bandlimited signals for trans- 
mission over a channel with additive white Gaussian noise, the signals to be 
received by a memoryless correlation detector. A signal waveform is found 
which allows communication at the Nyquist rate without inter symbol 
interference and with 1.3 db degradation compared to an optimum com- 
munication system. Other waveforms, consisting of the sum of a few pro- 
late spheroidal functions, are also investigated. 

I. INTRODUCTION 

111 the reception of serial binary data transmitted over a noisy band- 
limited channel, errors result from the combined effects of intersymbol 
interference and noise. Minimization of the error rate involves ap- 
propriate design of both the transmitted signal and the method of 
detection, taking into account the effects of both causes of degradation. 

Nyquist has shown how bandlimited signals may be designed so as 
to eliminate intersymbol interference when detection is accomplished 
by periodic instantaneous sampling. 1 Sunde has shown that optimum 
performance over a channel with white Gaussian noise is achieved when 
the shaping is divided equally between the transmitter and receiver. 2 
Tufts has developed a technique of long memory detection which elimi- 
nates intersymbol interference and optimizes noise performance subject 
to that constraint, for an arbitrary transmitted signal. 3 Kurz and Trabka 
have studied the design of signals for transmission in the presence of 
nonwhite noise without the problem of intersymbol interference. ' 



* This paper is based on pails of a thesis accepted by the faculty of the Grad- 
uate Division of the School of Engineering and Science of New York University 
in partial fulfillment of the requirements for the degree of Doctor of Engineering 
Science. 
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This paper discusses the design of bandlimited signals for communica- 
tion in the presence of white Gaussian noise, when the detector is a 
memoryless correlator. Memoryless correlation is a widely used sub- 
optimum means of detection. It will be shown in Section III that we can 
communicate without intersymbol interference at the Nyquist rate 
using memoryless correlation. In Section IV we investigate another 
form of signaling for communication with memoryless correlation. 
Here signals are chosen which do not eliminate intersymbol interference, 
but lead to low error probability for the most adverse message sequence. 

II. PRELIMINARIES 

In serial binary transmission, the nth binary digit of the message is 
transmitted by sending either s (t — nT) or Si(t — nT). We will as- 
sume that the a priori probabilities of so and Si are 1/2 and that all dig- 
its are independent. The transmitted information rate is therefore 
1/7' bits per second. 

If the signal is perturbed by additive white Gaussian noise, the 
optimum detector is well known to be a simple correlator if s (t) and 
si(t) are time limited to an interval of length T. Such a detector 
chooses so if and only if 

f v(t)s (t - nT)dt - i f s 2 (t - nT)dt 

> J v(t)si(t - nT)dt -if s, 2 U - nT)dt 

where v(t) is the received signal and the integration is taken over the 
interval of length T. 

A polar signal leads to minimum error probability: 7 

*(t) = Si(t) =f(t). 

The correlation detector then chooses s if and only if 



f v(t)f(t - nT)dt > 0. 



If f(t) is not time limited to an interval of length T, as is inevitable 
if it is bandlimited, then the memoryless correlator is a suboptinnun 
detector because it does not make use of the signal energy outside the 
interval. An infinite memory correlator or, equivalently, a matched 
filter and sampler, is the optimum detector, provided that intersymbol 
interference can be eliminated. The memoryless correlator, however, 



BANDLIMITED SIGNALS 237 

has found extensive practical application. With proper choice of /(*), 
the degradation as compared with optimum detection need not be too 
large. 

Aein and Hancock have shown that some improvement of the memory- 
less correlator can be obtained in the presence of intersymbol inter- 
ference by modifying the correlating function. 8 However, this pro- 
cedure is sensitive to amplitude variations of both the signal and the 
noise, whereas the simple correlation detector is not. We will therefore 
use the simple correlator and seek to minimize error probability through 
the choice of f(t). 

We will shift the time axis so that the origin is in the center of the 
bit to be detected, and assume that an infinite number of bits has been 
transmitted both before and after the bit currently being detected. 

v(t) = T,a*f(t + kT) +n(t) 

where a k = ±1 and n(t) is a member function of a stationary Gaussian 
random process with autocorrelation N[S(t)]/2. The one-sided spectral 
density of the noise is therefore N. 

Since we are using a simple correlation detector, a will be chosen as 

fT/2 

{ ao } = sgn / v(t)f(t)dt 

J- Til 

where 

Sgn x = |f| • 
The choice of ao when 

I " v(t)f(t)dt = 

J- Til 

is not important, since this event occurs with zero probability. 

Q = C~ v(t)f(t)dt 

J- 772 

is a linear functional of a Gaussian process and is therefore itself nor- 
mally distributed for a given sequence [a*]. Its expected value is 

E(Q) = E a k f ' fit + kT)f(t)dt, 

which may be written as 
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E{Q) = a d 2 + Io t p/ 



where 



and 



pT/2 

d 2 = f\t)dt (1) 

J— Til 



p« = T2 [ T ~f(t + kT)f(t)dt. (2) 



d 2 J-T/2 

The variance of Q is 

Var (Q) = | c/ 2 . 
The probability density of Q is therefore 

p(Q) = v^i exp [" m {Q ~ florf2 ~ § 0M ^'] • 

We may now calculate the probability of error as 

p(e ' * = +1) = vm £ exp Erw (0 - " ! - S W)! ] rfQ 
p(e ' "° = _1) = v^£ exp [~^i- {Q + ,e ~ 5 aW ' )! ] " Q - 

These expressions reduce to 

p(e | a = +1) = - erfcl ^= (l + g a fc p*)J 

p(e | a = -1) =2 erfc 7/^ C 1 ~~ 21 a * P *^ 
where 

2 r 00 

erfc (x) = — 7= / e~ l dt. 

VT •'x 

The maximum probability of error occurs when 

a% = —a sgn p k for all fc 5* (3) 

P»«-ierfc[^(l-El!*l)]. (4) 

It may be noted that if p fc = for all k ^ 0, then the probability of 



BANDLIMITED SIGNALS 239 

error is independent of the message sequence and is equal to 

\ erfc (d/VN). 

This is the case of no intersymbol interference, and the error probability 
is a monotone decreasing function of d. If intersymbol interference does 
exist, the error probability is greatest for the sequence (3) and is given 
by (4). The average error probability is tedious to calculate, but may 
be readily approximated. 9 

Equation (4) may be compared with the error probability for opti- 
mum detection 

p e = \ erfc {A/y/N) 
where 

a 2 = r 'f(t)dt). 

J— 00 

It is extremely desirable that the system perform error-free in the 
absence of noise, N = 0. Since 

lim p max = 0, if z2 | Pk | < 1 

= i, if ZIp* I = 1 

fc^O 

= 1, if E I P* I > 1» 

we will reject any system for which 2 I P* I = *> since m this case 
there will be some sequence of binary digits that cannot be received 
without error. 

III. SIGNALS WITHOUT INTERSYMBOL INTERFERENCE 

In order to avoid intersymbol interference with memoryless correla- 
tion detection, it is necessary that 

Pk = / f(t)f(t + kT)dt = d\k. (5) 

J_r/2 

We will seek bandlimited functions /(0 which satisfy (5) by using 
an unpublished method of H. O. Pollak. 

Let F(io) be the Fourier transform of f(t). If f(t) is bandlimited to 
I co I < co c , then 



fit) =i rF(a>)e> al dc 
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and 

T 

, r<* c /•»«. SU1 ( W _ X ) 7) 

Pk = ± F(u)F*(x) 7 ^- ****** dx. 



Let 



T 

sin (co — x) — 



(?(«) = ( °F*(x) , ^-dx. (6) 

J- w . IT ( CO — X 



Then 



P. =±- r F(u)G(a>)e ia "° T da>. 
4ir J- o>. 



We now divide the interval (— co c , w c ) into subintervals of length 

2w/T 

1 JV / .(2n+l)ir/T 

P* = ;r 2 / F(a,)G(u>)e j " kT da 

ZlT n=—N ■'(2n— l)ir/r 

where 



_^r ±/(„+*rw. 

JX J-irlT n=-N \ 1 / \ 



Pk = ±l £ FU + ^ ) G (co + ?£) 6^'dc. (7) 



Equation (7) indicates that the p*'s are the Fourier coefficients of the 
function 

BM = t^ P (. + 2 -2?) G (. + 5£) . . (8) 

Since p* = for all k 9^ 0, H(u) must be a constant independent of 
co. Using (5) and (7), we find that 

j£ r '(- + Tr) a '(- + ? r)- J *- (9) 

If cocT 7 ^ ir, then we may choose N = 0, and (9 ) reduces to 

*•(«)(?(«) = Td\ -|^to^J. (10) 

Equation (10) cannot be satisfied if F(co) = for any co in the in- 
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terval ( — ir/T,ir/T). Therefore intcrsymbol interference cannot be 
avoided if u c < v/T. 

Let ns now investigate the case co c = t/T. Substituting (6) into (10) 



T 

,/ r sin (w - x) - 

F(w) / F*(.r) ; r^dx = Td\ 

J-rlT 7T(cd — X) 



(ID 



In an unpublished work, Pedro Xowosad has proved that the quad- 
ratic integral equation (11) has a continuous, real, positive solution 

Equation (11) has been solved numerically by assuming an arbi- 
trary F (oi) and iteratively finding 

T 

TlT sin (w - x) - 

S-r-r - / Fn-iix) -. r—dx. 

Fn(a) J-tIt 7r(co — X) 

The resultant amplitude spectrum F(a>) is plotted in Fig. 1. The 
corresponding time function /(0 is plotted in Fig. 2. Since both /''(co) 
and/(/) are even functions, only the positive abscissas are shown. The 
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Fig. 1 — Spectrum of the signal which permits transmission without intersym- 
bol interference. 
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Fig. 2 — Bandlimited signal which permits transmission without intersymbol 
interference. 

signal has been normalized for unit total energy. This time function 
does indeed satisfy (5) with d 2 = 0.744. 

Digital communication using bandlimited signals and niemoryless 
correlation detection can therefore be achieved without intersymbol 
interference at the Nyquist rate, 1/T = u c /tt. The resultant degradation 
in the presence of white Gaussian noise, when compared with optimum 
detection of the signal sin u) c t/t, is —10 logio 0.744 = 1.3 db. 

A disadvantage which f(t) as shown in Fig. 2 shares with sin u c t/t 
is that 2/(£ + nT) docs not converge absolutely. Very large amplitudes 
may therefore be caused by certain sequences. If u c T > ir, then signals 
which converge more rapidly can easily be designed for detection by 
sampling. It is expected that solutions of (9) exist which also make such 
use of the additional available bandwidth. However, no such signals 
have as yet been found. 



IV. OTHER SIGNALS 

It is not at all necessary that intersymbol interference be eliminated 
in order to achieve reliable digital communication. For use with memory- 
less correlation, a signal with some intersymbol interference may very 
likely lead to a lower error probability than a signal with no intersymbol 
interference but with less of its energy in the principal time interval. 
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In this section we will drop the constraint of no intersymbol interfer- 
ence. The probability of error is therefore dependent on the message 
sequence. We will use a minimax type of criterion for designing the 
bandliinited signal. That is, we will attempt to minimize the proba- 
bility of error for the worst sequence. It is believed that the minimax 
criterion may be more realistic than an average error rate criterion, 
since the latter approach does not prevent the possibility of having 
some extremely sensitive message sequences. It is possible that such 
sensitive messages cannot be transmitted without error even over a 
noiseless channel. A further advantage of the minimax criterion is that 
it leads to a solution which does not require knowledge of the noise 
level. 

An additional constraint that will be imposed is that the signal am- 
plitude remain bounded for any message sequence. 

We will attempt to minimize p max as given by (4). From (4), p ma x 
is a monotone decreasing function of 

D = d(l - Z|p* |). (12) 

We can therefore satisfy the minimax criterion by maximizing the 
separation function, D. It is convenient to scale the amplitude of /(<) 
so that d *- 1. Such scaling, of course, affects the total energy of the 
signal. However, the quantity D/\/E remains invariant under such 
scaling, and we may accordingly maximize the quantity 



y/E 



i-Z 



*s*0 



( T ' 2 f(t)f(t+ kT)dt\ i (13) 

J-Tl-l IJ 



and the resultant /(/) may later be multiplied by the factor V ' A l jE 
in order to satisfy the fixed energy requirement. Here, A 2 is the required 
energy, while E is the energy of the scaled signal. 

It is also convenient to scale the time axis so that T = 2 and u> c = c, 
where the normalized bandwidth, c = \ u c T. Note that c = tt/2 cor- 
responds to transmission at the Xyquist rate. 

We will make use of the properties of the prolate spheroidal functions, 
\l/i(t), which are extensively discussed and plotted by Slepian, Landau 
and Pollak." 12 Some of these properties are 

[ MOMOdt = Sa 

J— 00 

j_ h(t)fi(t)di = A,5 U , 
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where X, is the (i + l)th largest eigenvalue of 

J-l T{t — V) 

ipi(t) is the eigenfunction corresponding to X, . Both \J/ t and X, depend 
on the parameter c. 

Since /(0 is a bandlimited function, it may be expressed as a series 
of prolate spheroidal functions" 

/(*) =T,yn*»(t). (14) 

n=0 

If we set 



A. 

7 "~ VZ' 



then 



M-te.*j-L. (is) 

„=o vx„ 

The functions ^ n (t)/\/x n are orthonormal over the interval ( — 1,1). 
/(/) can be expressed as a vector F = [/3 , ft , • • •], with orthonormal 
basis 

r ut) M) i 

LVVvV "J" 

The energy in the interval ( — 1,1) is equal to 

[j 2 {t)(U= FF' = ti + tf + ••• = 1, 
where F' is the transpose of F. The total energy is equal to 

E = f f{t)dt = ^ + f + • • • = FAF' 
J— ao Xo Xi 

where A is a diagonal matrix with elements A,-> = 5,-y/X,- . 

Since /(0 is bandlimited, f(t) in the interval (—1,1) determines /(/) 
for all time. 

fit + 2k) = £/3. X- - . 

„=o VA„ 

\{/ n (t + 2k)/-\/\ n can itself be expanded as 



where 



so that 
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+ n (t + 2k) _ A t hM 

■\Z\ n m=0 ^ VXm 

t mnk = -TrT ( tM+Jl + 2k)dt (16) 



/tt + 2fc) - £/3«Z^%=- 

n m V Am 

or, in matrix form, 

F, = FT k l (17) 

where the elements of T k are t ijk as given by (16). 

We can now express the intersymbol interference terms as 

f f(t)f(t + 2k)dt = FF k l = FT k F l . (18) 



Then 

1 - £ b k FT k F l 

D' = 



(19) 



Vfaf' 

where 

b k = sgn (FT k F'). 

We seek suboptimum solutions by confining F to il/ dimensions. 
That is, we will seek an optimum /* (0 of the form 

tM-Zem. (20) 

n =n VA„ 

Such an approach is justified if 

lim/j,(0 = f(0, 

Af-»oo 

the true optimum solution, and this convergence is sufficiently rapid. 
All vectors in the previous development are now M-dimensional and all 
square matrices are M X M. Note that (17) is no longer strictly cor- 
rect, but instead gives the projection of F k in the il/-dimensional space. 
Equations (18) and (19), however, remain valid. 

At this point we will introduce the constraint which requires that the 
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total signal amplitude remain bounded for any message sequence. 
This is highly desirable physically, because of the effects of inexact tim- 
ing and the technical impossibility of handling unbounded signals. 
For the worst sequence, 

w(0 = Zl/(* + 2*)l 

and we wish to constrain f(t) so that s max (0 remain bounded. We first 
express \f/„ (t) as a multiple of the radial prolate spheroidal function: 13 

Rn(t) 



fn(t) = 



K n 



where 



Since 11 



K n * = r R n \t)dt. 

•/-co 



Xn = — -Bn (1) 



we can also express K» as 



/C = 



T j R n \t)dt 

2cR n *(l) 



Then 



,3„ 



m-LkZk™ 



At) = E 






R n (t + 2k) \. 



For large \t\,R„(t) can be expressed asymptotically by 

R n (t) = (-l) n/2 ^ + 0(r 2 ), neven 

ct 

RM = (-l) (n+1)/2 55"* + 0(r 2 ), n odd. 
ct 



Let us examine 

s N (t) = E 



ft. 



n=0 K„-\/X n 



Rn(t + 2k) 



8 S 



(t) ^ E 



1*1— w 



BANDLIMITED SIGNALS 

t fin 



247 



„4* K n V\n 



0(k~ 2 ) 



+ E 

Hi-* 

00 

+ £ 

IfclmW 



fin sin c(< + 2fc) 



^e/ ' K n V\n C(t + 2fc) 



E (-D 



(n+l)/2. 



0n cos c(t + 2fc) 



n odd 



KnVxT c(< + 2fc) 



The last two series diverge, except for isolated values of c and t. 
Sufficient conditions for &w(0 to be bounded are therefore 



and 



E (-D" 2 

n even 

y (_D (n+1)/2 



KnV\n 

fin 

KnV\ n 



= 



= o. 



(21) 



(22) 



These equations confine F to an (M - 2) -dimensional subspace 
orthogonal to the two vectors 



V = 



r i — — i 



] 



FVo = FVi = 0. 



We can form an orthogonal matrix V in which the first two rows are 
CVo and KVi , and the remaining M - 2 rows are any vectors such 
that the M rows form an orthonormal set. The last M - 2 rows may, 
for example, be chosen by the Gram-Schmidt orthogonalization process. 
We may then form 

G = FV % = FV~ l 

since V 1 = V~ l for an orthogonal matrix. Due to the above constraints, 
the first two components of G, g t and g> = 0. Since V is an orthogonal 
transformation, GG' = FF = 1. 

We may also form matrices U k from T k . Since T k is used only in the 
quadratic form (18), we need only consider its symmetric component, 
TV, in which t ijk ' = t jik = |fe + t jik ). Then 

FT k F' = FT k F l = GVT k 'V'G l . 

Let U k = VTk'V*. lh is a symmetric matrix since it is congruent to 
Tk, a symmetric matrix. 
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FTmF* = GU k G'. 
If we also let 9 = VAV* 

1 - E b k 6U k 6* 

D' = **° ( 23 ) 

Vgqg 1 

We find the optimum M-dimensional signal /(0 by varying the unit- 
length, (M — 2) -dimensional vector G so as to maximize D given by 
(23), and then perform the inverse transformation and scaling. Note 
that if f(t) is constrained to be either an even or an odd function, only 
terms of even or odd n appear in (14), and only one of the constraints 
(21) or (22) is needed. 

The resultant f(t) is of the form (20). Landgrebe and Cooper have 
shown that the Fourier transform of \f/ n (t) is 1 



.14 



Mfc(oi-r*/g*(=), 



< c 



= 0, | CO I > C. 

Therefore the optimum f(t) may be generated by passing an impulse 
through a filter whose frequency response is 



Af-l / \ 

K E l-M. (-) . 

n=0 \C/ 



h(«) ^zrw»-, i co i < c 

n=0 VV 

= 0, | CO | > C. 

If M is reasonably small, H(cc) is well behaved, except at to = c, 
and may be readily approximated by a physically realizable filter. 

The optimum signals and their separation functions have been com- 
puted for several low-dimensional cases, each for several values of c. 
The total energy of the signals was set to unity in all cases. 

The simplest signal is a two-dimensional even or odd function. It is 
completely determined by its energy and constraint (21) or (22). 
Three such signals have been examined. The components of these three 
signals are ^o and ^2 , 1A1 and \f/ 3 , and \f/ 2 and ^4 , respectively. For all 
values of c, it was found that the first signal led to the highest value of 
the separation function D, while the third signal gave the lowest value 
of D. This result would be anticipated by energy considerations alone. 

The 70 and 72 components of the optimum two-dimensional signals 
are plotted in Fig. 3 as a function of the normalized bandwidth, c. 
The values of the separation function for these signals are shown in 
Fig. 5. 
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Fig. 3 — Components of optimum two-dimensional bandlimited signals. 

A three-dimensional signal may be formed with \f/ , ^ 2 and \p 4 com- 
ponents. One degree of freedom is available for adjusting the coefficients 
of these components so as to maximize D. The optimum coefficients for 
signals of this form are shown in Fig. 4. The resultant values of D are 
plotted in Fig. 5. It is seen that substantial improvement over the two- 
dimensional signal is obtained over a large range of c. 

A four-dimensional signal consisting of \f/ , ^i , ^ 2 and \f/ 3 components 
was also investigated. Constraints (21) and (22) and the energy re- 
quirement permit one degree of freedom in the signal design. It was 
found that no significant improvement over the two-dimensional signal 
could be obtained by using this form of signal. 

For an ideal signal which has all of its unit energy in the interval 
(—1,1), D = 1. Fig. 5 may be considered to be a comparison of the 
worst error probabilities of bandlimited signals to the error probability 
of an ideal signal. If D > 0, then the power of the bandlimited signal 
must be increased by -20 logio D db in order for its error probability 
for the worst sequence to be equal to the error probability of an ideal 
signal. 

It should be noted that for these signals, D < when c < t/2. 
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Fig. 4 — Components of optimum three-dimensional bandlimited signals. 

We must therefore transmit slower than the Nyquist rate in order to 
achieve error-free performance in the absence of noise. 



V. CONCLUSIONS 

Memoryless correlation is a suboptimum but useful method of de- 
tecting binary signals. With proper choice of the transmitted signal, 
the performance of a communication system using memoryless correla- 
tion can be made to be almost as good as that of an optimum system. 

Communication at the Nyquist rate without iiitersymbol interference 
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Fig. 5 — Separation functions for optimum two- and three-dimensional band- 
limited signals. 

using memoryless correlation detection is possible when the function 
shown in Fig. 2 is used as the transmitted signal. The resultant per- 
formance in the presence of noise is 1.3 db worse than that of an opti- 
mum system. 

Bandlimited signals may also be designed so as to lead to low error 
probabilities in spite of intersymbol interference. Signals consisting 
of linear combinations of a finite number of prolate spheroidal functions 
accomplish this purpose. These signals may be designed so as to remain 
bounded for all message sequences. 
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